Data Description:

https://olympus.greatlearning.in/courses/40612/files/4168350?module_item_id=1976416

You are provided with a dataset of images of plant seedlings at various stages of grown. Each image has a filename that is its unique id. The dataset comprises 12 plant species. The goal of the project is to create a classifier capable of determining a plant's species from a photo.

Link to the Kaggle project site: https://www.kaggle.com/c/plant-seedlings-classification/data?select=train

Context:

Can you differentiate a weed from a crop seedling? The ability to do so effectively can mean better crop yields and better stewardship of the environment. The Aarhus University Signal Processing group, in collaboration with University of Southern Denmark, has recently released a dataset containing images of unique plants belonging to 12 species at several growth stages

In [ ]:
# Get the data from google drive
from google.colab import drive
drive.mount('/content/drive')
Mounted at /content/drive

Overview of data

In [ ]:
# import standard libraries as per MLS2 Session example Brain Tumor problem

#Reading the training images from the path and labelling them into the given categories
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import cv2
import os
import seaborn as sns # for data visualization 
import tensorflow as tf
import keras
from sklearn.preprocessing import LabelEncoder
#ignoring warnings
import warnings
warnings.filterwarnings("ignore")

from sklearn.metrics import accuracy_score, confusion_matrix

from tensorflow.keras.models import Sequential #sequential api for sequential model 
from tensorflow.keras.layers import Dense, Dropout, Flatten #importing different layers 
from tensorflow.keras.layers import Conv2D, MaxPooling2D, BatchNormalization, Activation, Input, LeakyReLU,Activation
from tensorflow.keras import backend as K
from tensorflow.keras.utils import to_categorical #to perform one-hot encoding 
from keras.layers import Dense, Dropout, Flatten, Conv2D, MaxPool2D
from tensorflow.keras.optimizers import RMSprop,Adam #optimiers for optimizing the model
from keras.callbacks import EarlyStopping  #regularization method to prevent the overfitting
from keras.callbacks import ModelCheckpoint
from tensorflow.keras.models import Sequential, Model
from tensorflow.keras import losses, optimizers

# generate random integer values
from random import seed
from random import randint
# seed random number generator
seed(1)

# to suppress scientific notations
pd.set_option('display.float_format', lambda x: '%.3f' % x)

Load data and exploration

In [ ]:
image = np.load('/content/drive/MyDrive/Colab Notebooks/intro_to_compute_vision/project/images.npy')
labels = pd.read_csv('/content/drive/MyDrive/Colab Notebooks/intro_to_compute_vision/project/Labels.csv')
In [ ]:
image.shape
Out[ ]:
(4750, 128, 128, 3)

Observation

-4750 images with size 128x128 RGB (3 Channels).

In [ ]:
print(labels.nunique())
labels.value_counts()
Label    12
dtype: int64
Out[ ]:
Label                    
Loose Silky-bent             654
Common Chickweed             611
Scentless Mayweed            516
Small-flowered Cranesbill    496
Fat Hen                      475
Charlock                     390
Sugar beet                   385
Cleavers                     287
Black-grass                  263
Shepherds Purse              231
Maize                        221
Common wheat                 221
dtype: int64

Observations

  • DS don't have equal representation of all the classes of plant seedlings
In [ ]:
#plotting the pie chart 
lbls = ['Small-flowered Cranesbill', 'Fat Hen', 'Shepherds Purse', 'Common wheat', 'Common Chickweed', 'Charlock', 'Cleavers', 'Scentless Mayweed', 'Sugar beet', 'Maize', 'Black-grass', 'Loose Silky-bent']
count = []
for i in lbls:
    count.append(labels.Label[labels['Label'] == i ].count())
explode = (0, 0.1, 0, 0.1, 0, 0.1, 0, 0.1, 0, 0.1, 0, 0.1)
fig1, ax1 = plt.subplots(figsize=(10, 8))
ax1.pie(count, explode = explode, labels=lbls, autopct='%1.1f%%',
        shadow=True, startangle=90)
ax1.axis('equal')
plt.title(" % of different Plant Species ", size = 20)
plt.show()

Observations

  • Silky bent have max samples ~ 14%
  • Good representation of weeds (Common Chickweed, Scentless Mayweed) in image dataset.
In [ ]:
# Images from each class and their corresponding labels

data = [[image[i],labels.iloc[i].values[0]] for i in range(len(labels))]
df = pd.DataFrame(data)
class_names = labels['Label'].unique().tolist()

plt.figure(figsize=(20,20))
for i,class_name in enumerate(class_names):
    for img, label in df[df[1]==class_name].head(1).values.tolist():
        ax = plt.subplot(3, 4, i+1)
        plt.imshow(img)
        plt.title(label)
        plt.axis('off')

Observations

  • Black grass and silky bent seem to be tough to distinguish ... Need to take care of it later in the modelling.

EDA

In [ ]:
# Mean images of each class 
plt.figure(figsize=(20,20))
for i,class_name in enumerate(class_names):
    # calculate the average
    mean_img = np.mean(np.array((df[df[1]==class_name][0]).values.tolist()), axis = 0)
    # reshape it back to a matrix
    mean_img = mean_img.reshape((128,128,3)).astype(np.uint8)
    ax = plt.subplot(3, 4, i+1)
    plt.imshow(mean_img)
    plt.title(f'Average {class_name}')
    plt.axis('off')

Observations

  • Black Grass, Loose Silky there is good differentiation for green weed
  • Small-flowered cranesbill, Shepherds Purse have more wide green weed
In [ ]:
colors = ("red", "green", "blue")
channel_ids = (0, 1, 2)
plt.figure(figsize=(20,10))
# create the histogram plot, with three lines, one for
# each color

ax = plt.subplot(3, 4, i+1)

for i,class_name in enumerate(class_names):
    # calculate the average
    mean_img = np.mean(np.array((df[df[1]==class_name][0]).values.tolist()), axis = 0)
    # reshape it back to a matrix
    mean_img = mean_img.reshape((128,128,3)).astype(np.uint8)
    ax = plt.subplot(3, 4, i+1)
    for channel_id, c in zip(channel_ids, colors):
        histogram, bin_edges = np.histogram(
            mean_img[:, :, channel_id], bins=256, range=(0, 256)
        )
        ax.plot(bin_edges[0:-1], histogram, color=c)
        plt.title(f'Color Histogram :  {class_name}')

Observations

  • the RGB distribution matches with what I inferred in the mean plot. Small flowered and shephers purse have more more green differentation

EDA Conclusions

  • 4750 samplles with 128x128 with 3 channels (RGB)
  • DS don't equal representation in all classes
  • Sikly bent is max samples c.f to other samples. More overfitting in the model - need to account for that
  • black grasss and sikly bent seems to be difficult to differentiate
In [ ]:
# Data Pre-processing
In [ ]:
# remove the noise using cv2 Gaussian blurring
blurred_img = [cv2.GaussianBlur(image[i],(5,5),0) for i in range(len(image))]
In [ ]:
#normalization - as per MLS session
blurred_img_norm = np.array(blurred_img)/255.0
In [ ]:
#gray image
gray_img = [cv2.cvtColor(image[i],cv2.COLOR_BGR2GRAY) for i in range(len(image))]
blurred_gray_img = [cv2.blur(gray_img[i],(5,5)) for i in range(len(image))] 
blurred_gray_norm = np.array(blurred_gray_img)/255.0
In [ ]:
# vizualizing some random images with the labels
print("image before pre-processing")
plt.figure(figsize=(20,20))
idx=[]
for i in range(4):
    index = randint(0, 4749)
    idx.append(index)
    plt.subplot(2,2,i+1)
    plt.imshow(image[index])
    plt.title(labels.iloc[index].values[0])    
plt.show()
image before pre-processing
In [ ]:
# vizualizing some random images with the labels
print("After pre-processing")
plt.figure(figsize=(20,20))
for i, index in enumerate(idx):
    plt.subplot(2,2,i+1)
    plt.imshow(blurred_img[index], cmap = 'gray')
    plt.title(labels.iloc[index].values[0])    
plt.show()
After pre-processing

Let us process gray image - Gaussing and Normalization (pre and post)

In [ ]:
# vizualizing some random Gray images with the labels
print("Gausian/Normalization - Pre")
plt.figure(figsize=(20,20))
arr=[]
for i in range(4):
    index = randint(0, 4749)
    arr.append(index)
    plt.subplot(2,2,i+1)
    plt.imshow(gray_img[index], cmap ='gray')
    plt.title(labels.iloc[index].values[0])    
plt.show()
Gausian/Normalization - Pre
In [ ]:
# vizualizing some random images with the labels
print("Gaussian/Normalization - Post")
plt.figure(figsize=(20,20))
for i, index in enumerate(idx):
    plt.subplot(2,2,i+1)
    plt.imshow(blurred_gray_img[index], cmap = 'gray')
    plt.title(labels.iloc[index].values[0])    
plt.show()
Gaussian/Normalization - Post

Training and Test data

In [ ]:
"""
Algo:
choose color blurred image
retain image as few plant images (from eda) are hard to distinguish in gray images

"""
i = 0
X_train = []
y_train = []

X_test = []
y_test = []

idx = np.random.permutation(len(image))
x,y = np.array(blurred_img_norm)[idx], np.array(labels['Label'])[idx] #taking the normalized blurred img as input


for images,label in zip(x, y):
    i+=1
    if i < len(image)*.7:
        X_train.append(images)
        y_train.append(label)
    else:
        X_test.append(images)
        y_test.append(label)
X_train= np.array(X_train)
X_test = np.array(X_test)
print(X_train.shape)
print(X_test.shape)

# image reshaping 
X_train = X_train.reshape(-1,128,128,3)
X_test = X_test.reshape(-1,128,128,3)
(3324, 128, 128, 3)
(1426, 128, 128, 3)
In [ ]:
# vizualizing training images with the labels
print("Visualize training images with labels")
plt.figure(figsize=(20,20))
for i in range(25):
    index = randint(0, 3324)
    plt.subplot(5,5,i+1)
    plt.imshow(X_train[index])
    plt.title(y_train[index])
    plt.tight_layout()
plt.show()
Visualize training images with labels

Label encoding

In [ ]:
# convert categorical seedlings into labels
label_encoder = LabelEncoder()
y_train = label_encoder.fit_transform(y_train)
y_test = label_encoder.transform(y_test)
categories = list(label_encoder.classes_)

Hot Encoder

In [ ]:
encoded = to_categorical(y_train)
y_train_e=encoded
encoded_test = to_categorical(y_test)
y_test_e=encoded_test

New shape of DS

In [ ]:
print(X_train.shape)
print(X_test.shape)

# image reshaping 
X_train = X_train.reshape(-1,128,128,3)
X_test = X_test.reshape(-1,128,128,3)
(3324, 128, 128, 3)
(1426, 128, 128, 3)

Build the model

CNN - relu activation and adam optmizzer

In [ ]:
# Template from MLS Session 2
# CNN relu activation / adam optimizer

in_shape= X_train.shape[1:]  #shape of training dataset for the input to model

model1 = Sequential()
#
y_train=np.array(y_train)
model1.add(Conv2D(filters = 64, kernel_size = (5,5),padding = 'Same', 
                 activation ='relu', input_shape = in_shape))
model1.add(MaxPool2D(pool_size=(2,2)))

#
model1.add(Conv2D(filters = 128, kernel_size = (3,3),padding = 'Same', 
                 activation ='relu'))
model1.add(MaxPool2D(pool_size=(2,2), strides=(2,2)))

#
model1.add(Conv2D(filters = 128, kernel_size = (3,3),padding = 'Same', 
                 activation ='relu'))
model1.add(MaxPool2D(pool_size=(2,2), strides=(2,2)))

#
model1.add(Conv2D(filters = 128, kernel_size = (2,2),padding = 'Same', 
                 activation ='relu'))
model1.add(MaxPool2D(pool_size=(2,2), strides=(2,2)))


#
model1.add(Conv2D(filters = 256, kernel_size = (2,2),padding = 'Same', 
                 activation ='relu'))
model1.add(MaxPool2D(pool_size=(2,2), strides=(2,2)))


# 
model1.add(Flatten())
model1.add(Dense(1024, activation = "relu"))

model1.add(Dense(12, activation = "softmax"))

es = EarlyStopping(monitor='val_loss', mode='min', verbose=1, patience=20) #early stopping
mc = ModelCheckpoint('best_model1.h5', monitor='val_accuracy', mode='max', verbose=1, save_best_only=True) #model saving

model1.summary()
Model: "sequential"
_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 conv2d (Conv2D)             (None, 128, 128, 64)      4864      
                                                                 
 max_pooling2d (MaxPooling2D  (None, 64, 64, 64)       0         
 )                                                               
                                                                 
 conv2d_1 (Conv2D)           (None, 64, 64, 128)       73856     
                                                                 
 max_pooling2d_1 (MaxPooling  (None, 32, 32, 128)      0         
 2D)                                                             
                                                                 
 conv2d_2 (Conv2D)           (None, 32, 32, 128)       147584    
                                                                 
 max_pooling2d_2 (MaxPooling  (None, 16, 16, 128)      0         
 2D)                                                             
                                                                 
 conv2d_3 (Conv2D)           (None, 16, 16, 128)       65664     
                                                                 
 max_pooling2d_3 (MaxPooling  (None, 8, 8, 128)        0         
 2D)                                                             
                                                                 
 conv2d_4 (Conv2D)           (None, 8, 8, 256)         131328    
                                                                 
 max_pooling2d_4 (MaxPooling  (None, 4, 4, 256)        0         
 2D)                                                             
                                                                 
 flatten (Flatten)           (None, 4096)              0         
                                                                 
 dense (Dense)               (None, 1024)              4195328   
                                                                 
 dense_1 (Dense)             (None, 12)                12300     
                                                                 
=================================================================
Total params: 4,630,924
Trainable params: 4,630,924
Non-trainable params: 0
_________________________________________________________________
In [ ]:
# use adam optimizer

optimizer = Adam(learning_rate=0.001, beta_1=0.9, beta_2=0.999) #adam optimizer
model1.compile(optimizer = optimizer , loss = "categorical_crossentropy", metrics=["accuracy"])
epochs = 200  
batch_size = 64


history=model1.fit(X_train, 
          y_train_e,  #It expects integers because of the sparse_categorical_crossentropy loss function
          epochs=30, #number of iterations over the entire dataset to train on
          batch_size=64,validation_split=0.20,callbacks=[es,mc],use_multiprocessing=True)#number of samples per gradient update for training
Epoch 1/30
42/42 [==============================] - ETA: 0s - loss: 2.3887 - accuracy: 0.1760
Epoch 00001: val_accuracy improved from -inf to 0.24511, saving model to best_model1.h5
42/42 [==============================] - 13s 69ms/step - loss: 2.3887 - accuracy: 0.1760 - val_loss: 2.3024 - val_accuracy: 0.2451
Epoch 2/30
41/42 [============================>.] - ETA: 0s - loss: 1.7854 - accuracy: 0.3857
Epoch 00002: val_accuracy improved from 0.24511 to 0.43910, saving model to best_model1.h5
42/42 [==============================] - 2s 46ms/step - loss: 1.7833 - accuracy: 0.3855 - val_loss: 1.6682 - val_accuracy: 0.4391
Epoch 3/30
41/42 [============================>.] - ETA: 0s - loss: 1.4417 - accuracy: 0.5065
Epoch 00003: val_accuracy improved from 0.43910 to 0.50677, saving model to best_model1.h5
42/42 [==============================] - 2s 48ms/step - loss: 1.4395 - accuracy: 0.5073 - val_loss: 1.4355 - val_accuracy: 0.5068
Epoch 4/30
41/42 [============================>.] - ETA: 0s - loss: 1.1547 - accuracy: 0.5938
Epoch 00004: val_accuracy improved from 0.50677 to 0.61203, saving model to best_model1.h5
42/42 [==============================] - 2s 46ms/step - loss: 1.1537 - accuracy: 0.5942 - val_loss: 1.2183 - val_accuracy: 0.6120
Epoch 5/30
41/42 [============================>.] - ETA: 0s - loss: 0.9381 - accuracy: 0.6719
Epoch 00005: val_accuracy improved from 0.61203 to 0.63158, saving model to best_model1.h5
42/42 [==============================] - 2s 46ms/step - loss: 0.9410 - accuracy: 0.6706 - val_loss: 1.1609 - val_accuracy: 0.6316
Epoch 6/30
41/42 [============================>.] - ETA: 0s - loss: 0.8427 - accuracy: 0.7043
Epoch 00006: val_accuracy improved from 0.63158 to 0.64060, saving model to best_model1.h5
42/42 [==============================] - 2s 45ms/step - loss: 0.8415 - accuracy: 0.7052 - val_loss: 1.1778 - val_accuracy: 0.6406
Epoch 7/30
41/42 [============================>.] - ETA: 0s - loss: 0.6882 - accuracy: 0.7527
Epoch 00007: val_accuracy improved from 0.64060 to 0.71729, saving model to best_model1.h5
42/42 [==============================] - 2s 45ms/step - loss: 0.6882 - accuracy: 0.7510 - val_loss: 0.9124 - val_accuracy: 0.7173
Epoch 8/30
41/42 [============================>.] - ETA: 0s - loss: 0.5588 - accuracy: 0.8060
Epoch 00008: val_accuracy did not improve from 0.71729
42/42 [==============================] - 2s 41ms/step - loss: 0.5583 - accuracy: 0.8063 - val_loss: 0.9601 - val_accuracy: 0.7173
Epoch 9/30
41/42 [============================>.] - ETA: 0s - loss: 0.4813 - accuracy: 0.8346
Epoch 00009: val_accuracy did not improve from 0.71729
42/42 [==============================] - 2s 42ms/step - loss: 0.4811 - accuracy: 0.8349 - val_loss: 1.0102 - val_accuracy: 0.7008
Epoch 10/30
41/42 [============================>.] - ETA: 0s - loss: 0.3455 - accuracy: 0.8750
Epoch 00010: val_accuracy improved from 0.71729 to 0.74887, saving model to best_model1.h5
42/42 [==============================] - 2s 46ms/step - loss: 0.3453 - accuracy: 0.8751 - val_loss: 1.0417 - val_accuracy: 0.7489
Epoch 11/30
41/42 [============================>.] - ETA: 0s - loss: 0.2930 - accuracy: 0.8986
Epoch 00011: val_accuracy did not improve from 0.74887
42/42 [==============================] - 2s 42ms/step - loss: 0.2913 - accuracy: 0.8992 - val_loss: 1.1560 - val_accuracy: 0.7323
Epoch 12/30
41/42 [============================>.] - ETA: 0s - loss: 0.3141 - accuracy: 0.8868
Epoch 00012: val_accuracy improved from 0.74887 to 0.75338, saving model to best_model1.h5
42/42 [==============================] - 2s 45ms/step - loss: 0.3119 - accuracy: 0.8879 - val_loss: 1.0152 - val_accuracy: 0.7534
Epoch 13/30
41/42 [============================>.] - ETA: 0s - loss: 0.2377 - accuracy: 0.9146
Epoch 00013: val_accuracy improved from 0.75338 to 0.76391, saving model to best_model1.h5
42/42 [==============================] - 2s 47ms/step - loss: 0.2380 - accuracy: 0.9135 - val_loss: 1.0708 - val_accuracy: 0.7639
Epoch 14/30
41/42 [============================>.] - ETA: 0s - loss: 0.1918 - accuracy: 0.9310
Epoch 00014: val_accuracy improved from 0.76391 to 0.77444, saving model to best_model1.h5
42/42 [==============================] - 2s 45ms/step - loss: 0.1929 - accuracy: 0.9304 - val_loss: 1.0063 - val_accuracy: 0.7744
Epoch 15/30
41/42 [============================>.] - ETA: 0s - loss: 0.1347 - accuracy: 0.9497
Epoch 00015: val_accuracy improved from 0.77444 to 0.77594, saving model to best_model1.h5
42/42 [==============================] - 2s 45ms/step - loss: 0.1340 - accuracy: 0.9500 - val_loss: 1.1892 - val_accuracy: 0.7759
Epoch 16/30
41/42 [============================>.] - ETA: 0s - loss: 0.0900 - accuracy: 0.9680
Epoch 00016: val_accuracy improved from 0.77594 to 0.78346, saving model to best_model1.h5
42/42 [==============================] - 2s 46ms/step - loss: 0.0896 - accuracy: 0.9680 - val_loss: 1.4084 - val_accuracy: 0.7835
Epoch 17/30
41/42 [============================>.] - ETA: 0s - loss: 0.1320 - accuracy: 0.9489
Epoch 00017: val_accuracy did not improve from 0.78346
42/42 [==============================] - 2s 42ms/step - loss: 0.1338 - accuracy: 0.9481 - val_loss: 1.3827 - val_accuracy: 0.7564
Epoch 18/30
41/42 [============================>.] - ETA: 0s - loss: 0.1845 - accuracy: 0.9284
Epoch 00018: val_accuracy did not improve from 0.78346
42/42 [==============================] - 2s 42ms/step - loss: 0.1827 - accuracy: 0.9293 - val_loss: 1.1618 - val_accuracy: 0.7744
Epoch 19/30
41/42 [============================>.] - ETA: 0s - loss: 0.0830 - accuracy: 0.9699
Epoch 00019: val_accuracy did not improve from 0.78346
42/42 [==============================] - 2s 41ms/step - loss: 0.0857 - accuracy: 0.9695 - val_loss: 1.3774 - val_accuracy: 0.7353
Epoch 20/30
41/42 [============================>.] - ETA: 0s - loss: 0.1429 - accuracy: 0.9512
Epoch 00020: val_accuracy did not improve from 0.78346
42/42 [==============================] - 2s 41ms/step - loss: 0.1423 - accuracy: 0.9511 - val_loss: 1.4676 - val_accuracy: 0.7203
Epoch 21/30
41/42 [============================>.] - ETA: 0s - loss: 0.0980 - accuracy: 0.9668
Epoch 00021: val_accuracy did not improve from 0.78346
42/42 [==============================] - 2s 42ms/step - loss: 0.0992 - accuracy: 0.9665 - val_loss: 1.2193 - val_accuracy: 0.7744
Epoch 22/30
41/42 [============================>.] - ETA: 0s - loss: 0.0670 - accuracy: 0.9802
Epoch 00022: val_accuracy did not improve from 0.78346
42/42 [==============================] - 2s 42ms/step - loss: 0.0663 - accuracy: 0.9804 - val_loss: 1.4179 - val_accuracy: 0.7729
Epoch 23/30
41/42 [============================>.] - ETA: 0s - loss: 0.0470 - accuracy: 0.9836
Epoch 00023: val_accuracy did not improve from 0.78346
42/42 [==============================] - 2s 42ms/step - loss: 0.0475 - accuracy: 0.9835 - val_loss: 1.4291 - val_accuracy: 0.7714
Epoch 24/30
41/42 [============================>.] - ETA: 0s - loss: 0.0964 - accuracy: 0.9699
Epoch 00024: val_accuracy did not improve from 0.78346
42/42 [==============================] - 2s 42ms/step - loss: 0.0959 - accuracy: 0.9703 - val_loss: 1.4689 - val_accuracy: 0.7068
Epoch 25/30
41/42 [============================>.] - ETA: 0s - loss: 0.1314 - accuracy: 0.9607
Epoch 00025: val_accuracy did not improve from 0.78346
42/42 [==============================] - 2s 42ms/step - loss: 0.1329 - accuracy: 0.9605 - val_loss: 1.2429 - val_accuracy: 0.7714
Epoch 26/30
41/42 [============================>.] - ETA: 0s - loss: 0.0278 - accuracy: 0.9901
Epoch 00026: val_accuracy did not improve from 0.78346
42/42 [==============================] - 2s 42ms/step - loss: 0.0276 - accuracy: 0.9902 - val_loss: 1.4301 - val_accuracy: 0.7744
Epoch 27/30
41/42 [============================>.] - ETA: 0s - loss: 0.0185 - accuracy: 0.9931
Epoch 00027: val_accuracy improved from 0.78346 to 0.79398, saving model to best_model1.h5
42/42 [==============================] - 2s 46ms/step - loss: 0.0184 - accuracy: 0.9932 - val_loss: 1.5092 - val_accuracy: 0.7940
Epoch 00027: early stopping
In [ ]:
print(history.history.keys())
# summarize history for accuracy
plt.plot(history.history['accuracy'])
plt.plot(history.history['val_accuracy'])
plt.title('model accuracy')
plt.ylabel('accuracy')
plt.xlabel('epoch')
plt.legend(['train', 'test'], loc='upper left')
plt.show()
dict_keys(['loss', 'accuracy', 'val_loss', 'val_accuracy'])
In [ ]:
#evaluation on test set
model1.evaluate(X_test,y_test_e)
45/45 [==============================] - 1s 12ms/step - loss: 1.3766 - accuracy: 0.7882
Out[ ]:
[1.37660551071167, 0.7882187962532043]
In [ ]:
# Test Prediction 
y_test_pred = model1.predict(X_test)
y_test_pred_classes = np.argmax(y_test_pred, axis=1)
y_test_pred_prob = np.max(y_test_pred, axis=1)
In [ ]:
accuracy_score(np.array(y_test), y_test_pred_classes)
Out[ ]:
0.788218793828892
In [ ]:
matrix = confusion_matrix(np.array(y_test), y_test_pred_classes)

# Confusion matrix normalized per category true value
cf_matrix_n1 = np.round(matrix/np.sum(matrix, axis=1), 2)
plt.figure(figsize=(10,10))
sns.heatmap(cf_matrix_n1,xticklabels=categories, yticklabels=categories, annot=True)
Out[ ]:
<matplotlib.axes._subplots.AxesSubplot at 0x7faf49ff1710>

Observations

  • model overfit (high variance in accuracy)

Improve model - add dropout

In [ ]:
# Add dropout 
es = EarlyStopping(monitor='val_loss', mode='min', verbose=1, patience=200)
mc = ModelCheckpoint('best_model2.h5', monitor='val_accuracy', mode='max', verbose=1, save_best_only=True)
in_shape= X_train.shape[1:]
model_dropout = Sequential()
#
y_train=np.array(y_train)
model_dropout.add(Conv2D(filters = 64, kernel_size = (5,5),padding = 'Same', 
                 activation ='relu', input_shape = in_shape))
model_dropout.add(MaxPool2D(pool_size=(2,2)))
model_dropout.add(Dropout(0.25))
#
model_dropout.add(Conv2D(filters = 128, kernel_size = (3,3),padding = 'Same', 
                 activation ='relu'))
model_dropout.add(MaxPool2D(pool_size=(2,2), strides=(2,2)))
model_dropout.add(Dropout(0.25))
#
model_dropout.add(Conv2D(filters = 128, kernel_size = (3,3),padding = 'Same', 
                 activation ='relu'))
model_dropout.add(MaxPool2D(pool_size=(2,2), strides=(2,2)))
model_dropout.add(Dropout(0.3))
#
model_dropout.add(Conv2D(filters = 128, kernel_size = (2,2),padding = 'Same', 
                 activation ='relu'))
model_dropout.add(MaxPool2D(pool_size=(2,2), strides=(2,2)))
model_dropout.add(Dropout(0.3))

#
model_dropout.add(Conv2D(filters = 256, kernel_size = (2,2),padding = 'Same', 
                 activation ='relu'))
model_dropout.add(MaxPool2D(pool_size=(2,2), strides=(2,2)))
model_dropout.add(Dropout(0.3))

# 
model_dropout.add(Flatten())
model_dropout.add(Dense(1024, activation = "relu"))
model_dropout.add(Dropout(0.5))
model_dropout.add(Dense(12, activation = "softmax"))
In [ ]:
model_dropout.summary()
Model: "sequential_1"
_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 conv2d_5 (Conv2D)           (None, 128, 128, 64)      4864      
                                                                 
 max_pooling2d_5 (MaxPooling  (None, 64, 64, 64)       0         
 2D)                                                             
                                                                 
 dropout (Dropout)           (None, 64, 64, 64)        0         
                                                                 
 conv2d_6 (Conv2D)           (None, 64, 64, 128)       73856     
                                                                 
 max_pooling2d_6 (MaxPooling  (None, 32, 32, 128)      0         
 2D)                                                             
                                                                 
 dropout_1 (Dropout)         (None, 32, 32, 128)       0         
                                                                 
 conv2d_7 (Conv2D)           (None, 32, 32, 128)       147584    
                                                                 
 max_pooling2d_7 (MaxPooling  (None, 16, 16, 128)      0         
 2D)                                                             
                                                                 
 dropout_2 (Dropout)         (None, 16, 16, 128)       0         
                                                                 
 conv2d_8 (Conv2D)           (None, 16, 16, 128)       65664     
                                                                 
 max_pooling2d_8 (MaxPooling  (None, 8, 8, 128)        0         
 2D)                                                             
                                                                 
 dropout_3 (Dropout)         (None, 8, 8, 128)         0         
                                                                 
 conv2d_9 (Conv2D)           (None, 8, 8, 256)         131328    
                                                                 
 max_pooling2d_9 (MaxPooling  (None, 4, 4, 256)        0         
 2D)                                                             
                                                                 
 dropout_4 (Dropout)         (None, 4, 4, 256)         0         
                                                                 
 flatten_1 (Flatten)         (None, 4096)              0         
                                                                 
 dense_2 (Dense)             (None, 1024)              4195328   
                                                                 
 dropout_5 (Dropout)         (None, 1024)              0         
                                                                 
 dense_3 (Dense)             (None, 12)                12300     
                                                                 
=================================================================
Total params: 4,630,924
Trainable params: 4,630,924
Non-trainable params: 0
_________________________________________________________________
In [ ]:
optimizer = Adam(learning_rate=0.001, beta_1=0.9, beta_2=0.999)
model_dropout.compile(optimizer = optimizer , loss = "categorical_crossentropy", 
                      metrics=["accuracy"])
epochs = 200  
batch_size = 64


history2=model_dropout.fit(X_train, 
          y_train_e,  
          epochs=30, 
          batch_size=64,validation_split=0.20,callbacks=[es,mc],
          use_multiprocessing=True)
Epoch 1/30
42/42 [==============================] - ETA: 0s - loss: 2.4389 - accuracy: 0.1324
Epoch 00001: val_accuracy improved from -inf to 0.12180, saving model to best_model2.h5
42/42 [==============================] - 226s 5s/step - loss: 2.4389 - accuracy: 0.1324 - val_loss: 2.4408 - val_accuracy: 0.1218
Epoch 2/30
42/42 [==============================] - ETA: 0s - loss: 2.4192 - accuracy: 0.1369
Epoch 00002: val_accuracy did not improve from 0.12180
42/42 [==============================] - 219s 5s/step - loss: 2.4192 - accuracy: 0.1369 - val_loss: 2.4289 - val_accuracy: 0.1218
Epoch 3/30
42/42 [==============================] - ETA: 0s - loss: 2.4158 - accuracy: 0.1376
Epoch 00003: val_accuracy did not improve from 0.12180
42/42 [==============================] - 227s 5s/step - loss: 2.4158 - accuracy: 0.1376 - val_loss: 2.4289 - val_accuracy: 0.1218
Epoch 4/30
42/42 [==============================] - ETA: 0s - loss: 2.3875 - accuracy: 0.1783
Epoch 00004: val_accuracy improved from 0.12180 to 0.26165, saving model to best_model2.h5
42/42 [==============================] - 222s 5s/step - loss: 2.3875 - accuracy: 0.1783 - val_loss: 2.2025 - val_accuracy: 0.2617
Epoch 5/30
42/42 [==============================] - ETA: 0s - loss: 1.9813 - accuracy: 0.3193
Epoch 00005: val_accuracy improved from 0.26165 to 0.35038, saving model to best_model2.h5
42/42 [==============================] - 222s 5s/step - loss: 1.9813 - accuracy: 0.3193 - val_loss: 1.8982 - val_accuracy: 0.3504
Epoch 6/30
42/42 [==============================] - ETA: 0s - loss: 1.6562 - accuracy: 0.4096
Epoch 00006: val_accuracy improved from 0.35038 to 0.36842, saving model to best_model2.h5
42/42 [==============================] - 222s 5s/step - loss: 1.6562 - accuracy: 0.4096 - val_loss: 1.8062 - val_accuracy: 0.3684
Epoch 7/30
42/42 [==============================] - ETA: 0s - loss: 1.4700 - accuracy: 0.4806
Epoch 00007: val_accuracy improved from 0.36842 to 0.40451, saving model to best_model2.h5
42/42 [==============================] - 221s 5s/step - loss: 1.4700 - accuracy: 0.4806 - val_loss: 1.6483 - val_accuracy: 0.4045
Epoch 8/30
42/42 [==============================] - ETA: 0s - loss: 1.3381 - accuracy: 0.5378
Epoch 00008: val_accuracy improved from 0.40451 to 0.40602, saving model to best_model2.h5
42/42 [==============================] - 220s 5s/step - loss: 1.3381 - accuracy: 0.5378 - val_loss: 1.6941 - val_accuracy: 0.4060
Epoch 9/30
 4/42 [=>............................] - ETA: 3:09 - loss: 1.3125 - accuracy: 0.5117
In [34]:
print(history2.history.keys())
# summarize history for accuracy
plt.plot(history2.history['accuracy'])
plt.plot(history2.history['val_accuracy'])
plt.title('model accuracy')
plt.ylabel('accuracy')
plt.xlabel('epoch')
plt.legend(['train', 'test'], loc='upper left')
plt.show()
dict_keys(['loss', 'accuracy', 'val_loss', 'val_accuracy'])
In [36]:
#evaluate on test set
model_dropout.evaluate(X_test,y_test_e)
45/45 [==============================] - 0s 10ms/step - loss: 0.6109 - accuracy: 0.8247
Out[36]:
[0.6109350919723511, 0.8246844410896301]
In [37]:
matrix = confusion_matrix(np.array(y_test), y_test_pred_classes)
# Confusion matrix normalized per category true value
cf_matrix_n1 = np.round(matrix/np.sum(matrix, axis=1), 2)
plt.figure(figsize=(10,10))
sns.heatmap(cf_matrix_n1,xticklabels=categories, yticklabels=categories, annot=True)
Out[37]:
<matplotlib.axes._subplots.AxesSubplot at 0x7faf4c0af5d0>

Observations

  • Variance less, accurance increase

Model improve - activation fn, leaky relu, adam optimizer

In [38]:
class conv_Layers:
   
  def __init__(self, nfilters, kernel_size, stride=1, 
               pool_size=2, leakyrelu_slope=0.1, dropc=0.0, bnorm=False):
    self.nfilters = nfilters
    self.kernel_size = kernel_size
    self.stride = stride
    self.pool_size = pool_size
    self.leakyrelu_slope = leakyrelu_slope
    self.dropfrac = dropc
    self.bnorm = bnorm
  
  def __call__(self, x):
    x = Conv2D(self.nfilters, kernel_size=self.kernel_size, 
               strides=self.stride, padding='same')(x)
    x = LeakyReLU(self.leakyrelu_slope)(x)
    if (self.dropfrac > 0.0): 
      x = Dropout(self.dropfrac)(x)
    if (self.bnorm):
      x = BatchNormalization()(x)
    x = MaxPool2D(self.pool_size)(x)
    return x

class dense_Layers:

  def __init__(self, nunits, leakyrelu_slope=0.1, dropd=0.0, bnorm=False):
    self.nunits = nunits
    self.leakyrelu_slope = leakyrelu_slope 
    self.dropfrac = dropd
    self.bnorm = bnorm

  def __call__(self, x):
    x = Dense(self.nunits)(x)
    x = LeakyReLU(self.leakyrelu_slope)(x)
    if (self.dropfrac > 0.0):
      x = Dropout(self.dropfrac)(x)
    if (self.bnorm):
      x = BatchNormalization()(x)
    return x
In [39]:
def LNmodel(in_shape, conv_filters, dense_filters, kernel_size, num_classes, lr,
            stride=1, pool_size=2, leakyrelu_slope=0.1, dropc=0.0, dropd=0.0, bnorm=False):

  in_shape = X_train.shape[1:]
  i = Input(shape=in_shape)
  for ncl, nconvfilters in enumerate(conv_filters):
    if (ncl==0):
      x = conv_Layers(nconvfilters, kernel_size,
                      stride, pool_size, leakyrelu_slope, dropc, bnorm)(i)
    else:
      x = conv_Layers(nconvfilters, kernel_size,
                      stride, pool_size, leakyrelu_slope, dropc, bnorm)(x)

  x = Flatten()(x)

  for ndl, ndunits in enumerate(dense_filters):
    x = dense_Layers(ndunits, leakyrelu_slope, dropd, bnorm)(x)

  x = Dense(num_classes, activation='softmax')(x)

  ln_model  = Model(inputs=i, outputs=x)
  adam = optimizers.Adam(learning_rate=lr)
  ln_model.compile(loss='categorical_crossentropy', optimizer=adam, metrics=['accuracy'])
  return ln_model
In [40]:
lr = 0.001
kernelsize = 3
in_shape= X_train.shape[1:]
model = LNmodel(in_shape, [64,128,256], [1024,256], kernelsize, 12, lr,
                    stride=1, pool_size=2, leakyrelu_slope=0.1, dropc=0.25,
                    dropd=0.5, bnorm=False)
model.summary()
Model: "model"
_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 input_1 (InputLayer)        [(None, 128, 128, 3)]     0         
                                                                 
 conv2d_10 (Conv2D)          (None, 128, 128, 64)      1792      
                                                                 
 leaky_re_lu (LeakyReLU)     (None, 128, 128, 64)      0         
                                                                 
 dropout_6 (Dropout)         (None, 128, 128, 64)      0         
                                                                 
 max_pooling2d_10 (MaxPoolin  (None, 64, 64, 64)       0         
 g2D)                                                            
                                                                 
 conv2d_11 (Conv2D)          (None, 64, 64, 128)       73856     
                                                                 
 leaky_re_lu_1 (LeakyReLU)   (None, 64, 64, 128)       0         
                                                                 
 dropout_7 (Dropout)         (None, 64, 64, 128)       0         
                                                                 
 max_pooling2d_11 (MaxPoolin  (None, 32, 32, 128)      0         
 g2D)                                                            
                                                                 
 conv2d_12 (Conv2D)          (None, 32, 32, 256)       295168    
                                                                 
 leaky_re_lu_2 (LeakyReLU)   (None, 32, 32, 256)       0         
                                                                 
 dropout_8 (Dropout)         (None, 32, 32, 256)       0         
                                                                 
 max_pooling2d_12 (MaxPoolin  (None, 16, 16, 256)      0         
 g2D)                                                            
                                                                 
 flatten_2 (Flatten)         (None, 65536)             0         
                                                                 
 dense_4 (Dense)             (None, 1024)              67109888  
                                                                 
 leaky_re_lu_3 (LeakyReLU)   (None, 1024)              0         
                                                                 
 dropout_9 (Dropout)         (None, 1024)              0         
                                                                 
 dense_5 (Dense)             (None, 256)               262400    
                                                                 
 leaky_re_lu_4 (LeakyReLU)   (None, 256)               0         
                                                                 
 dropout_10 (Dropout)        (None, 256)               0         
                                                                 
 dense_6 (Dense)             (None, 12)                3084      
                                                                 
=================================================================
Total params: 67,746,188
Trainable params: 67,746,188
Non-trainable params: 0
_________________________________________________________________
In [42]:
es = EarlyStopping(monitor='val_loss', mode='min', verbose=1, patience=200)
mc = ModelCheckpoint('best_model3.h5', monitor='val_accuracy', mode='max', verbose=1, save_best_only=True)

history3 = model.fit(X_train, y_train_e, 
                                  validation_split=0.2,
                                  verbose=1, batch_size=64,
                                  shuffle=True, epochs=60,callbacks=[es,mc])
Epoch 1/60
42/42 [==============================] - ETA: 0s - loss: 2.4773 - accuracy: 0.2114
Epoch 00001: val_accuracy improved from -inf to 0.34436, saving model to best_model3.h5
42/42 [==============================] - 7s 148ms/step - loss: 2.4773 - accuracy: 0.2114 - val_loss: 2.0173 - val_accuracy: 0.3444
Epoch 2/60
41/42 [============================>.] - ETA: 0s - loss: 1.7590 - accuracy: 0.3979
Epoch 00002: val_accuracy did not improve from 0.34436
42/42 [==============================] - 3s 69ms/step - loss: 1.7563 - accuracy: 0.3975 - val_loss: 2.2957 - val_accuracy: 0.3188
Epoch 3/60
41/42 [============================>.] - ETA: 0s - loss: 1.4421 - accuracy: 0.4981
Epoch 00003: val_accuracy improved from 0.34436 to 0.35038, saving model to best_model3.h5
42/42 [==============================] - 5s 125ms/step - loss: 1.4391 - accuracy: 0.4983 - val_loss: 2.1895 - val_accuracy: 0.3504
Epoch 4/60
41/42 [============================>.] - ETA: 0s - loss: 1.1900 - accuracy: 0.5880
Epoch 00004: val_accuracy did not improve from 0.35038
42/42 [==============================] - 3s 77ms/step - loss: 1.1894 - accuracy: 0.5889 - val_loss: 2.4903 - val_accuracy: 0.3414
Epoch 5/60
41/42 [============================>.] - ETA: 0s - loss: 0.9708 - accuracy: 0.6791
Epoch 00005: val_accuracy did not improve from 0.35038
42/42 [==============================] - 3s 71ms/step - loss: 0.9722 - accuracy: 0.6788 - val_loss: 4.1143 - val_accuracy: 0.1955
Epoch 6/60
41/42 [============================>.] - ETA: 0s - loss: 0.8645 - accuracy: 0.7107
Epoch 00006: val_accuracy did not improve from 0.35038
42/42 [==============================] - 3s 71ms/step - loss: 0.8636 - accuracy: 0.7108 - val_loss: 3.5364 - val_accuracy: 0.2737
Epoch 7/60
41/42 [============================>.] - ETA: 0s - loss: 0.6885 - accuracy: 0.7732
Epoch 00007: val_accuracy did not improve from 0.35038
42/42 [==============================] - 3s 69ms/step - loss: 0.6905 - accuracy: 0.7728 - val_loss: 3.8310 - val_accuracy: 0.2586
Epoch 8/60
41/42 [============================>.] - ETA: 0s - loss: 0.5275 - accuracy: 0.8182
Epoch 00008: val_accuracy did not improve from 0.35038
42/42 [==============================] - 3s 70ms/step - loss: 0.5276 - accuracy: 0.8187 - val_loss: 3.4028 - val_accuracy: 0.2902
Epoch 9/60
41/42 [============================>.] - ETA: 0s - loss: 0.5207 - accuracy: 0.8281
Epoch 00009: val_accuracy did not improve from 0.35038
42/42 [==============================] - 3s 70ms/step - loss: 0.5211 - accuracy: 0.8278 - val_loss: 3.5974 - val_accuracy: 0.3143
Epoch 10/60
41/42 [============================>.] - ETA: 0s - loss: 0.4202 - accuracy: 0.8521
Epoch 00010: val_accuracy did not improve from 0.35038
42/42 [==============================] - 3s 69ms/step - loss: 0.4202 - accuracy: 0.8511 - val_loss: 4.0849 - val_accuracy: 0.2752
Epoch 11/60
41/42 [============================>.] - ETA: 0s - loss: 0.3399 - accuracy: 0.8853
Epoch 00011: val_accuracy did not improve from 0.35038
42/42 [==============================] - 3s 68ms/step - loss: 0.3400 - accuracy: 0.8849 - val_loss: 3.6476 - val_accuracy: 0.3398
Epoch 12/60
41/42 [============================>.] - ETA: 0s - loss: 0.2466 - accuracy: 0.9093
Epoch 00012: val_accuracy did not improve from 0.35038
42/42 [==============================] - 3s 68ms/step - loss: 0.2453 - accuracy: 0.9101 - val_loss: 4.8310 - val_accuracy: 0.2992
Epoch 13/60
41/42 [============================>.] - ETA: 0s - loss: 0.2122 - accuracy: 0.9253
Epoch 00013: val_accuracy improved from 0.35038 to 0.43910, saving model to best_model3.h5
42/42 [==============================] - 5s 126ms/step - loss: 0.2108 - accuracy: 0.9255 - val_loss: 2.8623 - val_accuracy: 0.4391
Epoch 14/60
41/42 [============================>.] - ETA: 0s - loss: 0.1872 - accuracy: 0.9322
Epoch 00014: val_accuracy did not improve from 0.43910
42/42 [==============================] - 3s 69ms/step - loss: 0.1881 - accuracy: 0.9319 - val_loss: 3.5179 - val_accuracy: 0.3474
Epoch 15/60
41/42 [============================>.] - ETA: 0s - loss: 0.1544 - accuracy: 0.9451
Epoch 00015: val_accuracy did not improve from 0.43910
42/42 [==============================] - 3s 72ms/step - loss: 0.1549 - accuracy: 0.9455 - val_loss: 5.8213 - val_accuracy: 0.2902
Epoch 16/60
41/42 [============================>.] - ETA: 0s - loss: 0.1428 - accuracy: 0.9527
Epoch 00016: val_accuracy did not improve from 0.43910
42/42 [==============================] - 3s 71ms/step - loss: 0.1429 - accuracy: 0.9530 - val_loss: 4.3445 - val_accuracy: 0.3383
Epoch 17/60
41/42 [============================>.] - ETA: 0s - loss: 0.1180 - accuracy: 0.9634
Epoch 00017: val_accuracy did not improve from 0.43910
42/42 [==============================] - 3s 70ms/step - loss: 0.1167 - accuracy: 0.9639 - val_loss: 3.4153 - val_accuracy: 0.4030
Epoch 18/60
41/42 [============================>.] - ETA: 0s - loss: 0.0944 - accuracy: 0.9688
Epoch 00018: val_accuracy did not improve from 0.43910
42/42 [==============================] - 3s 69ms/step - loss: 0.0977 - accuracy: 0.9673 - val_loss: 4.6146 - val_accuracy: 0.3429
Epoch 19/60
41/42 [============================>.] - ETA: 0s - loss: 0.1095 - accuracy: 0.9627
Epoch 00019: val_accuracy did not improve from 0.43910
42/42 [==============================] - 3s 77ms/step - loss: 0.1100 - accuracy: 0.9628 - val_loss: 3.3626 - val_accuracy: 0.4376
Epoch 20/60
41/42 [============================>.] - ETA: 0s - loss: 0.1162 - accuracy: 0.9596
Epoch 00020: val_accuracy improved from 0.43910 to 0.47519, saving model to best_model3.h5
42/42 [==============================] - 6s 153ms/step - loss: 0.1168 - accuracy: 0.9594 - val_loss: 3.1534 - val_accuracy: 0.4752
Epoch 21/60
41/42 [============================>.] - ETA: 0s - loss: 0.0747 - accuracy: 0.9798
Epoch 00021: val_accuracy improved from 0.47519 to 0.49173, saving model to best_model3.h5
42/42 [==============================] - 5s 129ms/step - loss: 0.0743 - accuracy: 0.9797 - val_loss: 3.3334 - val_accuracy: 0.4917
Epoch 22/60
41/42 [============================>.] - ETA: 0s - loss: 0.0964 - accuracy: 0.9710
Epoch 00022: val_accuracy did not improve from 0.49173
42/42 [==============================] - 3s 71ms/step - loss: 0.0958 - accuracy: 0.9714 - val_loss: 4.1647 - val_accuracy: 0.4526
Epoch 23/60
41/42 [============================>.] - ETA: 0s - loss: 0.0871 - accuracy: 0.9741
Epoch 00023: val_accuracy did not improve from 0.49173
42/42 [==============================] - 3s 70ms/step - loss: 0.0881 - accuracy: 0.9741 - val_loss: 5.6719 - val_accuracy: 0.3263
Epoch 24/60
41/42 [============================>.] - ETA: 0s - loss: 0.0810 - accuracy: 0.9718
Epoch 00024: val_accuracy did not improve from 0.49173
42/42 [==============================] - 3s 71ms/step - loss: 0.0816 - accuracy: 0.9718 - val_loss: 7.1904 - val_accuracy: 0.3158
Epoch 25/60
41/42 [============================>.] - ETA: 0s - loss: 0.0785 - accuracy: 0.9733
Epoch 00025: val_accuracy did not improve from 0.49173
42/42 [==============================] - 3s 71ms/step - loss: 0.0785 - accuracy: 0.9729 - val_loss: 4.4060 - val_accuracy: 0.4571
Epoch 26/60
41/42 [============================>.] - ETA: 0s - loss: 0.0892 - accuracy: 0.9748
Epoch 00026: val_accuracy did not improve from 0.49173
42/42 [==============================] - 3s 71ms/step - loss: 0.0893 - accuracy: 0.9748 - val_loss: 4.9274 - val_accuracy: 0.4135
Epoch 27/60
41/42 [============================>.] - ETA: 0s - loss: 0.0826 - accuracy: 0.9699
Epoch 00027: val_accuracy improved from 0.49173 to 0.50977, saving model to best_model3.h5
42/42 [==============================] - 6s 135ms/step - loss: 0.0816 - accuracy: 0.9703 - val_loss: 3.0579 - val_accuracy: 0.5098
Epoch 28/60
41/42 [============================>.] - ETA: 0s - loss: 0.0588 - accuracy: 0.9825
Epoch 00028: val_accuracy did not improve from 0.50977
42/42 [==============================] - 3s 71ms/step - loss: 0.0581 - accuracy: 0.9827 - val_loss: 4.2019 - val_accuracy: 0.4436
Epoch 29/60
41/42 [============================>.] - ETA: 0s - loss: 0.0418 - accuracy: 0.9878
Epoch 00029: val_accuracy did not improve from 0.50977
42/42 [==============================] - 3s 71ms/step - loss: 0.0415 - accuracy: 0.9880 - val_loss: 6.7713 - val_accuracy: 0.3083
Epoch 30/60
41/42 [============================>.] - ETA: 0s - loss: 0.0205 - accuracy: 0.9924
Epoch 00030: val_accuracy did not improve from 0.50977
42/42 [==============================] - 3s 69ms/step - loss: 0.0211 - accuracy: 0.9917 - val_loss: 4.2652 - val_accuracy: 0.4451
Epoch 31/60
41/42 [============================>.] - ETA: 0s - loss: 0.0479 - accuracy: 0.9809
Epoch 00031: val_accuracy did not improve from 0.50977
42/42 [==============================] - 3s 68ms/step - loss: 0.0474 - accuracy: 0.9812 - val_loss: 5.4624 - val_accuracy: 0.3865
Epoch 32/60
41/42 [============================>.] - ETA: 0s - loss: 0.0632 - accuracy: 0.9825
Epoch 00032: val_accuracy improved from 0.50977 to 0.58195, saving model to best_model3.h5
42/42 [==============================] - 5s 126ms/step - loss: 0.0627 - accuracy: 0.9827 - val_loss: 2.2692 - val_accuracy: 0.5820
Epoch 33/60
41/42 [============================>.] - ETA: 0s - loss: 0.1374 - accuracy: 0.9649
Epoch 00033: val_accuracy did not improve from 0.58195
42/42 [==============================] - 3s 69ms/step - loss: 0.1402 - accuracy: 0.9650 - val_loss: 3.8947 - val_accuracy: 0.4150
Epoch 34/60
41/42 [============================>.] - ETA: 0s - loss: 0.0814 - accuracy: 0.9756
Epoch 00034: val_accuracy did not improve from 0.58195
42/42 [==============================] - 3s 69ms/step - loss: 0.0804 - accuracy: 0.9759 - val_loss: 2.9276 - val_accuracy: 0.5263
Epoch 35/60
41/42 [============================>.] - ETA: 0s - loss: 0.0706 - accuracy: 0.9790
Epoch 00035: val_accuracy did not improve from 0.58195
42/42 [==============================] - 3s 69ms/step - loss: 0.0713 - accuracy: 0.9789 - val_loss: 4.4956 - val_accuracy: 0.4346
Epoch 36/60
41/42 [============================>.] - ETA: 0s - loss: 0.0505 - accuracy: 0.9848
Epoch 00036: val_accuracy did not improve from 0.58195
42/42 [==============================] - 3s 69ms/step - loss: 0.0503 - accuracy: 0.9846 - val_loss: 8.5844 - val_accuracy: 0.2692
Epoch 37/60
41/42 [============================>.] - ETA: 0s - loss: 0.0494 - accuracy: 0.9855
Epoch 00037: val_accuracy did not improve from 0.58195
42/42 [==============================] - 3s 69ms/step - loss: 0.0491 - accuracy: 0.9857 - val_loss: 9.2601 - val_accuracy: 0.2571
Epoch 38/60
41/42 [============================>.] - ETA: 0s - loss: 0.0650 - accuracy: 0.9825
Epoch 00038: val_accuracy did not improve from 0.58195
42/42 [==============================] - 3s 68ms/step - loss: 0.0648 - accuracy: 0.9823 - val_loss: 5.3465 - val_accuracy: 0.3925
Epoch 39/60
41/42 [============================>.] - ETA: 0s - loss: 0.0513 - accuracy: 0.9809
Epoch 00039: val_accuracy did not improve from 0.58195
42/42 [==============================] - 3s 69ms/step - loss: 0.0511 - accuracy: 0.9808 - val_loss: 5.9970 - val_accuracy: 0.3925
Epoch 40/60
41/42 [============================>.] - ETA: 0s - loss: 0.0350 - accuracy: 0.9893
Epoch 00040: val_accuracy did not improve from 0.58195
42/42 [==============================] - 3s 69ms/step - loss: 0.0362 - accuracy: 0.9883 - val_loss: 5.6634 - val_accuracy: 0.4316
Epoch 41/60
41/42 [============================>.] - ETA: 0s - loss: 0.0576 - accuracy: 0.9806
Epoch 00041: val_accuracy did not improve from 0.58195
42/42 [==============================] - 3s 75ms/step - loss: 0.0596 - accuracy: 0.9804 - val_loss: 4.4118 - val_accuracy: 0.5053
Epoch 42/60
41/42 [============================>.] - ETA: 0s - loss: 0.0798 - accuracy: 0.9832
Epoch 00042: val_accuracy did not improve from 0.58195
42/42 [==============================] - 3s 69ms/step - loss: 0.0793 - accuracy: 0.9831 - val_loss: 3.7381 - val_accuracy: 0.4677
Epoch 43/60
41/42 [============================>.] - ETA: 0s - loss: 0.0533 - accuracy: 0.9870
Epoch 00043: val_accuracy did not improve from 0.58195
42/42 [==============================] - 3s 69ms/step - loss: 0.0525 - accuracy: 0.9872 - val_loss: 4.5651 - val_accuracy: 0.4226
Epoch 44/60
41/42 [============================>.] - ETA: 0s - loss: 0.0442 - accuracy: 0.9882
Epoch 00044: val_accuracy did not improve from 0.58195
42/42 [==============================] - 3s 69ms/step - loss: 0.0439 - accuracy: 0.9883 - val_loss: 4.5725 - val_accuracy: 0.4827
Epoch 45/60
41/42 [============================>.] - ETA: 0s - loss: 0.0404 - accuracy: 0.9851
Epoch 00045: val_accuracy did not improve from 0.58195
42/42 [==============================] - 3s 69ms/step - loss: 0.0399 - accuracy: 0.9853 - val_loss: 7.1247 - val_accuracy: 0.3639
Epoch 46/60
41/42 [============================>.] - ETA: 0s - loss: 0.0408 - accuracy: 0.9867
Epoch 00046: val_accuracy did not improve from 0.58195
42/42 [==============================] - 3s 72ms/step - loss: 0.0404 - accuracy: 0.9868 - val_loss: 6.4179 - val_accuracy: 0.3504
Epoch 47/60
41/42 [============================>.] - ETA: 0s - loss: 0.0311 - accuracy: 0.9916
Epoch 00047: val_accuracy did not improve from 0.58195
42/42 [==============================] - 3s 73ms/step - loss: 0.0319 - accuracy: 0.9914 - val_loss: 4.6705 - val_accuracy: 0.4902
Epoch 48/60
41/42 [============================>.] - ETA: 0s - loss: 0.0203 - accuracy: 0.9935
Epoch 00048: val_accuracy did not improve from 0.58195
42/42 [==============================] - 3s 75ms/step - loss: 0.0201 - accuracy: 0.9936 - val_loss: 6.7372 - val_accuracy: 0.4526
Epoch 49/60
41/42 [============================>.] - ETA: 0s - loss: 0.0343 - accuracy: 0.9916
Epoch 00049: val_accuracy did not improve from 0.58195
42/42 [==============================] - 3s 71ms/step - loss: 0.0347 - accuracy: 0.9914 - val_loss: 4.5643 - val_accuracy: 0.4842
Epoch 50/60
41/42 [============================>.] - ETA: 0s - loss: 0.0434 - accuracy: 0.9851
Epoch 00050: val_accuracy did not improve from 0.58195
42/42 [==============================] - 3s 71ms/step - loss: 0.0432 - accuracy: 0.9853 - val_loss: 10.4053 - val_accuracy: 0.2556
Epoch 51/60
41/42 [============================>.] - ETA: 0s - loss: 0.0474 - accuracy: 0.9855
Epoch 00051: val_accuracy improved from 0.58195 to 0.59398, saving model to best_model3.h5
42/42 [==============================] - 5s 131ms/step - loss: 0.0472 - accuracy: 0.9853 - val_loss: 2.8482 - val_accuracy: 0.5940
Epoch 52/60
41/42 [============================>.] - ETA: 0s - loss: 0.0666 - accuracy: 0.9771
Epoch 00052: val_accuracy improved from 0.59398 to 0.68271, saving model to best_model3.h5
42/42 [==============================] - 5s 126ms/step - loss: 0.0661 - accuracy: 0.9774 - val_loss: 2.4331 - val_accuracy: 0.6827
Epoch 53/60
41/42 [============================>.] - ETA: 0s - loss: 0.0607 - accuracy: 0.9855
Epoch 00053: val_accuracy did not improve from 0.68271
42/42 [==============================] - 3s 69ms/step - loss: 0.0603 - accuracy: 0.9853 - val_loss: 4.0097 - val_accuracy: 0.4707
Epoch 54/60
41/42 [============================>.] - ETA: 0s - loss: 0.0416 - accuracy: 0.9878
Epoch 00054: val_accuracy did not improve from 0.68271
42/42 [==============================] - 3s 68ms/step - loss: 0.0414 - accuracy: 0.9880 - val_loss: 4.0582 - val_accuracy: 0.5459
Epoch 55/60
41/42 [============================>.] - ETA: 0s - loss: 0.0727 - accuracy: 0.9817
Epoch 00055: val_accuracy did not improve from 0.68271
42/42 [==============================] - 3s 68ms/step - loss: 0.0737 - accuracy: 0.9812 - val_loss: 4.1783 - val_accuracy: 0.5053
Epoch 56/60
41/42 [============================>.] - ETA: 0s - loss: 0.0456 - accuracy: 0.9874
Epoch 00056: val_accuracy did not improve from 0.68271
42/42 [==============================] - 3s 68ms/step - loss: 0.0451 - accuracy: 0.9876 - val_loss: 4.0039 - val_accuracy: 0.5158
Epoch 57/60
41/42 [============================>.] - ETA: 0s - loss: 0.0375 - accuracy: 0.9901
Epoch 00057: val_accuracy did not improve from 0.68271
42/42 [==============================] - 3s 69ms/step - loss: 0.0370 - accuracy: 0.9902 - val_loss: 3.7876 - val_accuracy: 0.5188
Epoch 58/60
41/42 [============================>.] - ETA: 0s - loss: 0.0316 - accuracy: 0.9905
Epoch 00058: val_accuracy did not improve from 0.68271
42/42 [==============================] - 3s 69ms/step - loss: 0.0321 - accuracy: 0.9902 - val_loss: 4.1896 - val_accuracy: 0.5218
Epoch 59/60
41/42 [============================>.] - ETA: 0s - loss: 0.0504 - accuracy: 0.9889
Epoch 00059: val_accuracy did not improve from 0.68271
42/42 [==============================] - 3s 68ms/step - loss: 0.0497 - accuracy: 0.9891 - val_loss: 4.3796 - val_accuracy: 0.4481
Epoch 60/60
41/42 [============================>.] - ETA: 0s - loss: 0.0299 - accuracy: 0.9912
Epoch 00060: val_accuracy did not improve from 0.68271
42/42 [==============================] - 3s 68ms/step - loss: 0.0297 - accuracy: 0.9914 - val_loss: 7.7047 - val_accuracy: 0.3564
In [43]:
print(history3.history.keys())
# summarize history for accuracy
plt.plot(history3.history['accuracy'])
plt.plot(history3.history['val_accuracy'])
plt.title('model accuracy')
plt.ylabel('accuracy')
plt.xlabel('epoch')
plt.legend(['train', 'test'], loc='upper left')
plt.show()
dict_keys(['loss', 'accuracy', 'val_loss', 'val_accuracy'])
In [44]:
# Test Prediction 
y_test_pred = model.predict(X_test)
y_test_pred_classes = np.argmax(y_test_pred, axis=1)
y_test_pred_prob = np.max(y_test_pred, axis=1)
In [46]:
matrix = confusion_matrix(np.array(y_test), y_test_pred_classes)

# Confusion matrix normalized per category true value
cf_matrix_n1 = np.round(matrix/np.sum(matrix, axis=1), 2)
plt.figure(figsize=(10,10))
sns.heatmap(cf_matrix_n1,xticklabels=categories, yticklabels=categories, annot=True)
Out[46]:
<matplotlib.axes._subplots.AxesSubplot at 0x7faeaa5553d0>

Categories

  • in this model, accuracy is better
  • this model performs c.f to the other 2 models

Eval the 3 models

Confusion matrix

In [55]:
#helper function
def confusion_matrix_plot(model):
  y_test_pred = model.predict(X_test)
  y_test_pred_classes = np.argmax(y_test_pred, axis=1)
  y_test_pred_prob = np.max(y_test_pred, axis=1)
  cf_matrix = confusion_matrix(np.array(y_test), y_test_pred_classes)
  # Confusion matrix normalized per category true value
  cf_matrix_n1 = np.round(cf_matrix/np.sum(cf_matrix, axis=1), 2)
  plt.figure(figsize=(10,10))
  sns.heatmap(cf_matrix_n1,xticklabels=categories, yticklabels=categories, annot=True)
In [56]:
confusion_matrix_plot(model1)
In [51]:
# first model evaluation on test set
model1.evaluate(X_test,y_test_e)
45/45 [==============================] - 0s 9ms/step - loss: 1.3766 - accuracy: 0.7882
Out[51]:
[1.37660551071167, 0.7882187962532043]
In [57]:
confusion_matrix_plot(model_dropout)
In [59]:
# CM for 2nd model
confusion_matrix_plot(model)
In [50]:
# 2nd model evaluation on test set
model_dropout.evaluate(X_test,y_test_e)
45/45 [==============================] - 0s 10ms/step - loss: 0.6109 - accuracy: 0.8247
Out[50]:
[0.6109350919723511, 0.8246844410896301]
In [58]:
# CM for 3rd model
confusion_matrix_plot(model)
In [49]:
#3rd model evaluation on test set
model.evaluate(X_test,y_test_e)
45/45 [==============================] - 1s 13ms/step - loss: 7.7888 - accuracy: 0.3485
Out[49]:
[7.78879451751709, 0.34852734208106995]
In [53]:
comparison_frame = pd.DataFrame({'Model':['first_model : CNN model','2nd_model : CNN Model (Dropout Included)',
                                          '3rd_model - Leaky Relu Activation and Adam Optimizer'], 'Training Accuracy':[0.992,0.905,0.986], 'Validation Accuray':[0.784,0.792,0.612], 'Test Set Accuray':[0.772,0.807,0.597], 'Remarks':['High Variance, Overfitted on Training Set','Improved and Reqularized Model','High Variance and Low Accuracy']}) 
comparison_frame
Out[53]:
Model Training Accuracy Validation Accuray Test Set Accuray Remarks
0 first_model : CNN model 0.992 0.784 0.772 High Variance, Overfitted on Training Set
1 2nd_model : CNN Model (Dropout Included) 0.905 0.792 0.807 Improved and Reqularized Model
2 3rd_model - Leaky Relu Activation and Adam Opt... 0.986 0.612 0.597 High Variance and Low Accuracy

Conclusions

  • 2nd model performs the best on accuracy

Predictions - pictorially

In [60]:
# Model2 to see the prediction
chk = [2,3,33,36,59]
plt.figure(figsize=(15,18))
for i,n  in enumerate(chk):
    plt.subplot(3,2,i+1)
    plt.imshow(X_test[n]) #plant seedlings
    ttl = '\n'+'Prediction: ' + str(categories[np.argmax(model_dropout.predict(X_test)[n])]) + '\n'+'Actual: ' + str(categories[y_test[n]])
    plt.title(ttl) 
    plt.tight_layout()   
plt.show()

Observations

  • 2nd model predicted all classifications correctly

Final notes

  • We have accuracy of 80% with 2nd model
  • In sample validation, it was successful
In [ ]: